Aspects of this disclosure are generally related to distributed data storage systems which can be used to maintain relatively large data sets and support a relatively large number of concurrent users. The basic building blocks of a distributed data storage system may include computing nodes such as storage engines or storage servers. The computing nodes provide access to data storage devices such as disk drives and flash drives. A plurality of computing nodes can be interconnected in order to function together as a cluster. A plurality of clusters can be interconnected by communication links to enable the clusters to function as a single distributed storage system.